Methods and systems for enhancing depth perception of a non-visible spectrum image of a scene

ABSTRACT

A method and system for providing depth perception to a two-dimensional (2D) representation of a given three-dimensional (3D) object within a 2D non-visible spectrum image of a scene is provided. The method comprises: capturing the 2D non-visible spectrum image at a capture time, by at least one non-visible spectrum sensor; obtaining 3D data regarding the given 3D object independently of the 2D non-visible spectrum image; generating one or more depth cues based on the 3D data; applying the depth cues on the 2D representation to generate a depth perception image that provides the depth perception to the 2D representation; and displaying the depth perception image.

TECHNICAL FIELD

The invention relates to methods and systems for enhancing a depthperception of a non-visible spectrum image of a scene.

BACKGROUND

Non-visible spectrum sensors (e.g., infrared sensors) can capture imagesthat are not in the visible spectrum, i.e. non-visible spectrum images,in a wide variety of applications. However, non-visible spectrum imageshave poor depth perception. That is, a non-visible spectrum image poorlyperceives three-dimensional (3D) features that are present within thescene that is displayed in the non-visible spectrum image.

Thus, there is a need in the art for new methods and systems forenhancing a depth perception of a non-visible spectrum image of a scene.

References considered to be relevant as background to the presentlydisclosed subject matter are listed below. Acknowledgement of thereferences herein is not to be inferred as meaning that these are in anyway relevant to the patentability of the presently disclosed subjectmatter.

U.S. Pat. No. 6,157,733 (“Swain”), published on Dec. 5, 2000, disclosesone or more monocular cues being extracted from an original image andcombined to enhance depth effect. An original image is acquired andsegmented into one or more objects. The objects are identified as beingeither in the foreground or the background, and an object of interest isidentified. One or more depth cues are then extracted from the originalimage, including shading, brightness, blur and occlusion. The depth cuesmay be in the form of one or more intermediate images having an improveddepth effect. The depth cues are then combined or applied to create animage with enhanced depth effect.

U.S. Patent Application Publication No. 2015/0208054 “Michot”),published on Jul. 23, 2015, discloses a method of generating a depth cuefor three dimensional video content. The method comprises the steps of(a) detecting three dimensional video content that will appear inobserver space when displayed; (b) identifying a reference projectionparameter; (c) estimating a location of a shadow that would be generatedby the detected content as a consequence of a light source emittinglight according to the reference projection parameter; and (d)projecting light content imitating a shadow to the estimated location tocoincide with display of the three dimensional video content. Alsodisclosed are a computer program product for carrying out a method ofgenerating a depth cue for three dimensional video content and anapparatus for generating a depth cue for three dimensional videocontent.

GENERAL DESCRIPTION

In accordance with a first aspect of the presently disclosed subjectmatter, there is provided a method for providing depth perception to atwo-dimensional (2D) representation of a given three-dimensional (3D)object within a 2D non-visible spectrum image of a scene, the methodcomprising: capturing the 2D non-visible spectrum image at a capturetime, by at least one non-visible spectrum sensor; obtainingthree-dimensional (3D) data regarding the given 3D object independentlyof the 2D non-visible spectrum image; generating one or more depth cuesbased on the 3D data; applying the depth cues on the 2D representationto generate a depth perception image that provides the depth perceptionto the 2D representation; and displaying the depth perception image.

In some cases, the 3D data is a priori data regarding coordinates of afixed coordinate system established in space that are associated withthe given 3D object, the a priori data being available prior to thecapture time, and wherein the depth cues are generated based on the apriori data and an actual position and orientation of the non-visiblespectrum sensor relative to the fixed coordinate system at the capturetime.

In some cases, the 3D data is one or more readings by an additionalsensor that is distinct from the non-visible spectrum sensor, andwherein the depth cues are generated based on the readings and a firstactual position and orientation of the non-visible spectrum sensor atthe capture time relative to a second actual position and orientation ofthe additional sensor at a second time of the readings.

In some cases, the 3D data is a priori data regarding coordinates of afixed coordinate system established in space that are associated withthe given 3D object, the a priori data being available prior to thecapture time, and wherein the depth cues are generated prior to thecapture time, based on the a priori data and an expected position andorientation of the non-visible spectrum sensor relative to the fixedcoordinate system at the capture time.

In some cases, the method further comprises: recording the 2Dnon-visible spectrum image to provide a recording of the 2D non-visiblespectrum image; wherein the depth cues are applied on the 2Drepresentation within the 2D non-visible spectrum image as recorded.

In some cases, the additional sensor is one of: a depth sensor or anactive 3D scanner.

In some cases, the active 3D scanner is a Light Detection and Ranging(LiDAR).

In some cases, the depth cues include one or more of the following: (a)one or more shadows; (b) a virtual object; or (c) contour lines.

In some cases, at least some of the shadows are generated by one or morevirtual light sources.

In some cases, the method further comprises: selecting one or moreselected light sources of the virtual light sources.

In some cases, the method further comprises: for at least one selectedlight source of the selected light sources, defining one or moreparameters of the at least one selected light source, the one or moreparameters including a position and an orientation of the at least oneselected light source.

In some cases, for at least one selected light source of the selectedlight sources, one or more parameters of the at least one selected lightsource are defined by a user, the one or more parameters including aposition and an orientation of the at least one selected light source.

In some cases, one or more selected light sources of the virtual lightsources are selected by a user, and the user defines one or moreparameters of the selected light sources, the one or more parametersincluding a position and an orientation of each selected light source ofthe selected light sources.

In some cases, at least some of the shadows are generated based on aknown position and orientation of an existing light source thatilluminates the scene at the capture time.

In some cases, the virtual object is distinguishable from the 2Drepresentation.

In accordance with a second aspect of the presently disclosed subjectmatter, there is provided a method for enhancing a depth perception of anon-visible spectrum image of a scene, the method comprising: capturingthe non-visible spectrum image at a capture time, by at least onenon-visible spectrum sensor, the non-visible spectrum image includingone or more objects; classifying one or more of the objects withoutderiving three-dimensional (3D) data from the non-visible spectrumimage, giving rise to one or more classified objects; generating one ormore depth cues based on one or more parameters associated with theclassified objects; applying the depth cues to the non-visible spectrumimage to generate an enhanced depth perception image having an enhanceddepth perception relative to the non-visible spectrum image; anddisplaying the enhanced depth perception image.

In some cases, the depth cues include one or more of the following: (a)one or more shadows; (b) one or more virtual objects that are based on acorresponding one or more physical objects that are of a known size; (c)haze; or (d) perspective.

In some cases, at least some of the shadows are generated by one or morevirtual light sources.

In some cases, at least some of the shadows are generated based on aknown position and orientation of an existing light source thatilluminates the scene at the capture time.

In accordance with a third aspect of the presently disclosed subjectmatter, there is provided a system for providing depth perception to atwo-dimensional (2D) representation of a given three-dimensional (3D)object within a 2D non-visible spectrum image of a scene, the systemcomprising: at least one non-visible spectrum sensor configured tocapture the 2D non-visible spectrum image at a capture time; and aprocessing circuitry configured to: obtain three-dimensional (3D) dataregarding the given 3D object independently of the 2D non-visiblespectrum image; generate one or more depth cues based on the 3D data;apply the depth cues on the 2D representation to generate a depthperception image that provides the depth perception relative to the 2Drepresentation; and display the depth perception image.

In some cases, the 3D data is a priori data regarding coordinates of afixed coordinate system established in space that are associated withthe given 3D object, the a priori data being available prior to thecapture time, and wherein the depth cues are applied based on the apriori data and an actual position and orientation of the non-visiblespectrum sensor relative to the fixed coordinate system at the capturetime.

In some cases, the 3D data is one or more readings by an additionalsensor that is distinct from the non-visible spectrum sensor, andwherein the depth cues are generated based on the readings and a firstactual position and orientation of the non-visible spectrum sensor atthe capture time relative to a second actual position and orientation ofthe additional sensor at a second time of the readings.

In some cases, the 3D data is a priori data regarding coordinates of afixed coordinate system established in space that are associated withthe given 3D object, the a priori data being available prior to thecapture time, and wherein the depth cues are generated prior to thecapture time, based on the a priori data and an expected position andorientation of the non-visible spectrum sensor relative to the fixedcoordinate system at the capture time.

In some cases, the processing circuitry is further configured to: recordthe 2D non-visible spectrum image to provide a recording of the 2Dnon-visible spectrum image; wherein the depth cues are applied on the 2Dnon-visible spectrum image as recorded.

In some cases, the additional sensor is one of: a depth sensor or anactive 3D scanner.

In some cases, the active 3D scanner is a Light Detection and Ranging(LiDAR).

In some cases, the depth cues include one or more of the following: (a)one or more shadows; (b) a virtual object; or (c) contour lines.

In some cases, at least some of the shadows are generated by one or morevirtual light sources.

In some cases, the processing circuitry is further configured to: selectone or more selected light sources of the virtual light sources.

In some cases, the processing circuitry is further configured to: for atleast one selected light source of the selected light sources, defineone or more parameters of the at least one selected light source, theone or more parameters including a position and an orientation of the atleast one selected light source.

In some cases, for at least one selected light source of the selectedlight sources, one or more parameters of the at least one selected lightsource are defined by a user of the system, the one or more parametersincluding a position and an orientation of the at least one selectedlight source.

In some cases, one or more selected light sources of the virtual lightsources are selected by a user of the system, and the user defines oneor more parameters of the selected light sources, the one or moreparameters including a position and an orientation of each selectedlight source of the selected light sources.

In some cases, at least some of the shadows are generated based on aknown position and orientation of an existing light source thatilluminates the scene at the capture time.

In some cases, the virtual object is distinguishable from the 2Drepresentation.

In accordance with a fourth aspect of the presently disclosed subjectmatter, there is provided a system for enhancing a depth perception of anon-visible spectrum image of a scene, the system comprising: at leastone non-visible spectrum sensor configured to capture the non-visiblespectrum image at a capture time, the non-visible spectrum imageincluding one or more objects; and a processing circuitry configured to:classify one or more of the objects without deriving three-dimensional(3)) data from the non-visible spectrum image, giving rise to one ormore classified objects; generate one or more depth cues based on one ormore parameters of the classified objects; apply the depth cues to thenon-visible spectrum image to generate an enhanced depth perceptionimage having an enhanced depth perception relative to the non-visiblespectrum image; and display the enhanced depth perception image.

In some cases, the depth cues include one or more of the following: (a)one or more shadows; (b) one or more virtual objects that are based on acorresponding one or more physical objects that are of a known size; (c)haze; or (d) perspective.

In some cases, at least some of the shadows are generated by one or morevirtual light sources.

In some cases, at least some of the shadows are generated based on aknown position and orientation of an existing light source thatilluminates the scene at the capture time.

In accordance with a fifth aspect of the presently disclosed subjectmatter, there is provided a non-transitory computer readable storagemedium having computer readable program code embodied therewith, thecomputer readable program code, executable by processing circuitry of acomputer to perform a method for providing depth perception to atwo-dimensional (2D) representation of a given three-dimensional (3D)object within a 2D non-visible spectrum image of a scene, the methodcomprising: capturing the 2D non-visible spectrum image at a capturetime, by at least one non-visible spectrum sensor; obtainingthree-dimensional (3D) data regarding the given 3D object independentlyof the 2D non-visible spectrum image; generating one or more depth cuesbased on the 3D data; applying the depth cues on the 2D representationto generate a depth perception image that provides the depth perceptionto the 2D representation; and displaying the depth perception image.

In accordance with a sixth aspect of the presently disclosed subjectmatter, there is provided a non-transitory computer readable storagemedium having computer readable program code embodied therewith, thecomputer readable program code, executable by processing circuitry of acomputer to perform a method for enhancing a depth perception of anon-visible spectrum image of a scene, the method comprising: capturingthe non-visible spectrum image at a capture time, by at least onenon-visible spectrum sensor, the non-visible spectrum image includingone or more objects; classifying one or more of the objects withoutderiving three-dimensional (3D) data from the non-visible spectrumimage, giving rise to one or more classified objects; generating one ormore depth cues based on one or more parameters associated with theclassified objects; applying the depth cues to the non-visible spectrumimage to generate an enhanced depth perception image having an enhanceddepth perception relative to the non-visible spectrum image; anddisplaying the enhanced depth perception image.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the presently disclosed subject matter and to seehow it may be carried out in practice, the subject matter will now bedescribed, by way of non-limiting examples only, with reference to theaccompanying drawings, in which:

FIG. 1 is a block diagram schematically illustrating an example of asystem for enhancing a depth perception of a non-visible spectrum imageof a scene, in accordance with the presently disclosed subject matter;and

FIG. 2 is a flowchart illustrating a first example of a sequence ofoperations for enhancing a depth perception of a non-visible spectrumimage of a scene, in accordance with the presently disclosed subjectmatter:

FIG. 3 is a flowchart illustrating a second example of a sequence ofoperations for enhancing a depth perception of a non-visible spectrumimage of a scene, in accordance with the presently disclosed subjectmatter;

FIG. 4 is a schematic diagram illustrating a schematic opticalinstrument for displaying an enhanced depth perception image of thescene, in accordance with the presently disclosed subject matter;

FIG. 5 is a schematic diagram illustrating an exploded view of anenhanced eyepiece of the schematic optical instrument, in accordancewith the presently disclosed subject matter;

FIG. 6 is a schematic diagram illustrating another perspective of theschematic optical instrument, in accordance with the presently disclosedsubject matter; and

FIG. 7 is an optical diagram illustrating optical components of theenhanced eyepiece, in accordance with the presently disclosed subjectmatter.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the presentlydisclosed subject matter. However, it will be understood by thoseskilled in the art that the presently disclosed subject matter may bepracticed without these specific details. In other instances, well-knownmethods, procedures, and components have not been described in detail soas not to obscure the presently disclosed subject matter.

In the drawings and descriptions set forth, identical reference numeralsindicate those components that are common to different embodiments orconfigurations.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “capturing”, “obtaining”,“generating”, “applying”. “displaying”, “recording”, “classifying” orthe like, include actions and/or processes, including, inter alia,actions and/or processes of a computer, that manipulate and/or transformdata into other data, said data represented as physical quantities, e.g.such as electronic quantities, and/or said data representing thephysical objects. The terms “computer”, “processor”, “processingcircuitry” and “controller” should be expansively construed to cover anykind of electronic device with data processing capabilities, including,by way of non-limiting example, a personal desktop/laptop computer, aserver, a computing system, a communication device, a smartphone, atablet computer, a smart television, a processor (e.g. digital signalprocessor (DSP), a microcontroller, a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), etc.), agroup of multiple physical machines sharing performance of varioustasks, virtual servers co-residing on a single physical machine, anyother electronic computing device, and/or any combination thereof.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to “one case”,“some cases”. “other cases” or variants thereof means that a particularfeature, structure or characteristic described in connection with theembodiment(s) is included in at least one embodiment of the presentlydisclosed subject matter. Thus the appearance of the phrase “one case”,“some cases”, “other cases” or variants thereof does not necessarilyrefer to the same embodiment(s).

It is appreciated that, unless specifically stated otherwise, certainfeatures of the presently disclosed subject matter, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the presently disclosed subject matter, which are, forbrevity, described in the context of a single embodiment, may also beprovided separately or in any suitable sub-combination.

In embodiments of the presently disclosed subject matter, fewer, moreand/or different stages than those shown in FIGS. 2 and 3 may beexecuted. In embodiments of the presently disclosed subject matter oneor more stages illustrated in FIGS. 2 and 3 may be executed in adifferent order and/or one or more groups of stages may be executedsimultaneously. FIGS. 1 and 4 to 7 illustrate a general schematic of thesystem architecture in accordance with embodiments of the presentlydisclosed subject matter. Each module in FIG. 1 can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in FIG. 1 may becentralized in one location or dispersed over more than one location. Inother embodiments of the presently disclosed subject matter, the systemmay comprise fewer, more, and/or different modules than those shown inFIG. 1 .

Any reference in the specification to a method should be applied mutatismutandis to a system capable of executing the method and should beapplied mutatis mutandis to a non-transitory computer readable mediumthat stores instructions that once executed by a computer result in theexecution of the method.

Any reference in the specification to a system should be applied mutatismutandis to a method that may be executed by the system and should beapplied mutatis mutandis to a non-transitory computer readable mediumthat stores instructions that may be executed by the system.

Any reference in the specification to a non-transitory computer readablemedium should be applied mutatis mutandis to a system capable ofexecuting the instructions stored in the non-transitory computerreadable medium and should be applied mutatis mutandis to method thatmay be executed by a computer that reads the instructions stored in thenon-transitory computer readable medium.

Attention is now drawn to FIG. 1 , a block diagram schematicallyillustrating an example of a system 100 for enhancing a depth perceptionof a non-visible spectrum image of a scene, in accordance with thepresently disclosed subject matter.

In accordance with the presently disclosed subject matter, system 100can be configured to include at least one non-visible spectrum sensor110. Non-visible spectrum sensor 110 can be configured to capture imagesthat are not in the visible spectrum. i.e. non-visible spectrum images.In some cases, non-visible spectrum sensor 110 can be configured tocapture infrared (IR) images, extremely high frequency (EHF) images ormillimeter-wave (MMW) radar images. The non-visible spectrum images thatare captured by non-visible spectrum sensor 110 generally have poordepth perception.

System 100 can further comprise or be otherwise associated with a datarepository 120 (e.g. a database, a storage system, a memory includingRead Only Memory—ROM, Random Access Memory—RAM, or any other type ofmemory, etc.) configured to store data. In some cases, the data storedcan include three-dimensional (3D) data of part or all of a scene thatis captured in a non-visible spectrum image, as detailed further herein,inter alia with reference to FIG. 2 . In some cases, data repository 120can be further configured to enable retrieval and/or update and/ordeletion of the stored data. It is to be noted that in some cases, datarepository 120 can be distributed.

System 100 can also be configured to include processing circuitry 130.Processing circuitry 130 can be one or more processing units (e.g.central processing units), microprocessors, microcontrollers (e.g.microcontroller units (MCUs)) or any other computing devices or modules,including multiple and/or parallel and/or distributed processing units,which are adapted to independently or cooperatively process data forcontrolling relevant system 100 resources and for enabling operationsrelated to system 100 resources.

Processing circuitry 130 can be configured to include a depth perceptionenhancement module 140. Depth perception enhancement module 140 can beconfigured to apply depth cues to a non-visible spectrum image of ascene to generate an enhanced depth perception image, as detailedfurther herein, inter alia with reference to FIGS. 2 to 7 .

In some cases, system 100 can be configured to include a synthetic imagesource (not shown in FIG. 1 ) for optically injecting the depth cuesonto the non-visible spectrum image of the scene to generate theenhanced depth perception image, as detailed further herein, inter aliawith reference to FIGS. 2 to 7 .

Attention is now drawn to FIG. 2 , a flowchart illustrating a firstexample of a sequence of operations 200 for enhancing a depth perceptionof a non-visible spectrum image of a scene, in accordance with thepresently disclosed subject matter.

In accordance with the presently disclosed subject matter, the at leastone non-visible spectrum sensor 110 can be configured to capture anon-visible spectrum image of a scene at a capture time. In some cases,the non-visible spectrum image can be a two-dimensional (2D) non-visiblespectrum image (block 204 t.

Processing circuitry 130 can be configured. e.g. using depth perceptionenhancement module 140, to obtain three-dimensional (3D) data of one ormore regions within the scene independently of the non-visible spectrumimage, the one or more regions comprising part or all of the scene. Insome cases, the 3D data can be regarding one or more given 3D objectsthat have a 2D representation within the 2D non-visible spectrum image(block 208).

In some cases, the 3D data can include a priori data that is availableto the system 100 (e.g., stored in the data repository 120) prior to thecapture time, the a priori data being associated with coordinates of afixed coordinate system established in space. The a priori data can be,for example, a stored terrain elevation model. In some cases, the 3Ddata can be a priori data regarding coordinates of a fixed coordinatesystem established in space that are associated with one or more given3D objects that have a 2D representation within the 2D non-visiblespectrum image.

Additionally, or alternatively, in some cases, at least some of the 3Ddata can be obtained from a depth sensor (not shown) that is distinctfrom non-visible spectrum sensor 110, based on one or more readings bythe depth sensor. In some cases, the 3D data obtained from the depthsensor can be 3D data regarding one or more given 3D objects that have a2D representation within the 2D non-visible spectrum image. In somecases, the depth sensor can be coupled to the non-visible spectrumsensor 110. Additionally, or as a further alternative, in some cases, atleast some of the 3D data can be a 3D map that is generated based onreadings obtained by an active 3D scanner (not shown) from at least onescan of one or more of the regions within the scene. In some cases, the3D map can include 3D data regarding one or more given 3D objects thathave a 2D representation within the 2D non-visible spectrum image. Insome cases, the active 3D scanner can be coupled to the non-visiblespectrum sensor 110. Additionally, or alternatively, in some cases, theactive 3D scanner and the non-visible spectrum sensor 110 can be mountedon a common body. In some cases, the active 3D scanner can be a LightDetection and Ranging (LiDAR).

Processing circuitry 130 can be further configured, e.g. using depthperception enhancement module 140, to generate one or more depth cuesbased on the 3D data (block 212).

In some cases, one or more of the depth cues can be generated based on apriori data, as defined above, and an actual position and orientation ofthe non-visible spectrum sensor 110 relative to the fixed coordinatesystem at the capture time. Additionally, or alternatively, in somecases, one or more of the depth cues can be generated prior to thecapture time, as defined above, based on a priori data, as definedabove, and an expected position and orientation of non-visible spectrumsensor 110 relative to the fixed coordinate system at the capture time.

Additionally, or as a further alternative, in some cases, one or more ofthe depth cues can be generated based on readings by an additionalsensor that is distinct from the non-visible spectrum sensor, forexample, a depth sensor, as defined above, or an active 3D scanner, asdefined above; a first actual position and orientation of thenon-visible spectrum sensor at the capture time relative to a secondactual position and orientation of the additional sensor at a secondtime of the readings by the additional sensor; and the actual positionand orientation of the non-visible spectrum sensor 110 at the capturetime.

In some cases, the non-visible spectrum image can capture one or moreobjects in the scene, if any (for example, a 2D representation of given3D objects in the scene can be captured by a 2D non-visible spectrumimage). In order to apply one or more depth cues to provide enhanceddepth perception for a given object of the objects in the scene, theobtained 3D data of the scene can be correlated with the given object.For example, if the 3D data is a terrain elevation model, a location ofthe given object relative to the terrain elevation model can bedetermined. This can be achieved by calculating a location (i.e.,position and orientation) of the non-visible spectrum sensor 110relative to the given object at the capture time, the location of thenon-visible spectrum sensor 110 relative to the terrain in the scene atthe capture time being known. As an additional example, if the 3D dataof the scene is obtained from readings of an additional sensor, i.e. adepth sensor or an active 3D scanner, a location of the given objectrelative to the 3D data of the scene can be determined by calculating alocation (i.e., position and orientation) of the non-visible spectrumsensor 110 relative to the given object at the capture time, and therelation between the location of the non-visible spectrum sensor 110 atthe capture time and the additional sensor at the time that readingsassociated with the given object are obtained by the additional sensor.

In some cases, certain objects in a scene can be considered to be ofhigher priority than other objects in the scene. In some cases, depthcues can be generated for the higher priority objects. For example, ifthe non-visible spectrum sensor 110 is mounted on an airborne platform(e.g., airplane, helicopter, drone, etc.), the objects in the scene thatare present along the flight path of the airborne platform can beconsidered to be of a higher priority, and depth cues can be generatedfor these objects.

Processing circuitry 130 can be configured. e.g. using depth perceptionenhancement module 140, to apply the depth cues to the non-visiblespectrum image to generate an enhanced depth perception image having anenhanced depth perception relative to the non-visible spectrum image. Insome cases, the depth cues can be applied on one or more 2Drepresentations, of a corresponding one or more given 3D objects, withinthe 2D non-visible spectrum image to generate a depth perception imagethat provides depth perception to the 2D representations (block 216).

In some cases, processing circuitry 130 can be configured to record thenon-visible spectrum image (e.g., the 2D non-visible spectrum image) toprovide a recorded non-visible spectrum image (e.g., a recording of the2D non-visible spectrum image), and to apply at least one of the depthcues to the recorded non-visible spectrum image (e.g., on the 2Drepresentations of the given 3D objects within the 2D non-visiblespectrum image as recorded), thereby giving rise to the enhanced depthperception image.

In some cases, the depth cues can be applied to the non-visible spectrumimage by overlaying the non-visible spectrum image with the depth cues(rather than modifying the non-visible spectrum image with the depthcues, e.g., merging the depth cues with the non-visible spectrum image).As a specific example, depth cues can be applied on a 2D representationof a given 3D object within a 2D non-visible spectrum image byoverlaying the depth cues on the 2D non-visible spectrum image. Theoverlaying of the depth cues on the non-visible spectrum image can beperformed, for example, by optically injecting the depth cues by asynthetic image source (not shown) and combining the depth cues with anunaltered non-visible spectrum image (i.e., overlaid on the non-visiblespectrum image), e.g. by an optical combiner, to display an enhanceddepth perception image to the user of the system 100, as detailedfurther herein, inter alia with reference to FIGS. 4 to 7 . In thismanner, the enhanced depth perception image of the scene can bedisplayed without latency.

In some cases, the depth cues can be overlaid onto the non-visiblespectrum image within an eyepiece of an optical instrument that displaysthe enhanced depth perception image to the user of the system 100, asdetailed further herein, inter alia with reference to FIGS. 4 to 7 . Insome cases, the optical instrument can be a night vision opticalinstrument.

In some cases, in which one or more depth cues are applied on a 2Drepresentation of a given 3D object within the 2D non-visible spectrumimage, the depth cues can be applied on the 2D representation by mergingthe depth cues with the 2D non-visible spectrum image.

Processing circuitry 130 can also be configured, e.g. using depthperception enhancement module 140, to display the enhanced depthperception image (block 220).

In some cases, the depth cues that are applied to (e.g., overlaid on ormerged with) the non-visible spectrum image can include one or moreshadows that are generated based on at least some of the 3D data, theshadows being configured to at least one of: (a) vary a brightness ofpixels within the enhanced depth perception image relative to thenon-visible spectrum image or (b) vary a color of pixels within theenhanced depth perception image relative to the non-visible spectrumimage. For example, one or more shadows can be applied on a 2Drepresentation of a given 3D object in a 2D non-visible spectrum imageto provide depth perception to the 2D representation.

In some cases, at least some of the shadows can be generated by one ormore virtual light sources. Additionally, or alternatively, in somecases, at least some of the shadows can be generated based on a knownposition and orientation of an existing light source (sun, moon, otherlight source, etc.) that illuminates the scene that is captured by thenon-visible spectrum sensor 110.

In some cases, in which at least some of the shadows are generated byone or more virtual light sources, at least some of the virtual lightsources can be selected by the processing circuitry 130, based on the 3Ddata. In some cases, processing circuitry 130 can further be configuredto define at least some of the parameters of a respective virtual lightsource, of the virtual light sources that it selects, based on the 3Ddata. Additionally, or alternatively, in some cases, a user can defineat least some of the parameters of a respective virtual light source, ofthe virtual light sources that are selected by processing circuitry 130,based on the 3D data. The parameters of a respective virtual lightsource can be, for example, a magnitude (e.g., intensity), a positionand an orientation, and/or other characteristics of the respectivevirtual light source.

In some cases, in which at least some of the shadows are generated byone or more virtual light sources, at least some of the virtual lightsources can be selected by a user, based on the 3D data. The user canalso define the parameters of the virtual light sources that he/sheselects, based on the 3D data.

In some cases, the depth cues that are applied to the non-visiblespectrum image (e.g., are overlaid on the non-visible spectrum image)can include one or more virtual objects, the virtual objects being basedon a corresponding one or more physical objects that are of a knownsize. Prior to applying the virtual objects to the non-visible spectrumimage, an actual size of each of the virtual objects can be determined,e.g. by processing circuitry 130, in accordance with a known size of thephysical object to which it corresponds, and a distance between thenon-visible spectrum sensor 110 and a location within the non-visiblespectrum image at which the respective virtual object is to be applied(i.e., overlaid).

In some cases, the depth cues that are applied on a 2D representation ofa given 3D object in a 2D non-visible spectrum image can include avirtual object that is a 3D representation of the given 3D object.

In some cases, processing circuitry 130 can be configured to define oneor more parameters that are associated with the virtual objects.Additionally, or alternatively, in some cases, a user can define one ormore parameters that are associated with the virtual objects. Theparameters that are associated with the virtual objects can be, forexample, a type(s) (e.g., car, building, tree, stationary, moving,etc.), a number, a location and/or a distribution of the virtualobjects.

In some cases, a moving virtual object can be added at a known distancefrom the non-visible spectrum sensor 110 to track movement in a locationof the non-visible spectrum sensor 110 with respect to the scene that iscaptured by the non-visible spectrum sensor 110.

Moreover, processing circuitry 130 can be configured to apply thevirtual objects to the non-visible spectrum image (e.g., merge thevirtual objects with the non-visible spectrum image, overlay the virtualobjects on the non-visible spectrum image, etc.), such that the virtualobjects in the displayed enhanced depth perception image aredistinguishable from real objects in the displayed enhanced depthperception image. In some cases, in which a virtual object that is a 3Drepresentation of a given 3D object is applied on a 2D representation ofthe given 3D object in the 2D non-visible spectrum image, the virtualobject can be applied such that the virtual object is distinguishablefrom the 2D representation.

In some cases, in which one or more depth cues are applied on a 2Drepresentation of a given 3D object within the 2D non-visible spectrumimage, the depth cues can include contour lines that provide a depthcontour on the 2D representation.

In some cases, the depth cues that are applied to the non-visiblespectrum image can include ha/e. Processing circuitry 130 can beconfigured to apply the haze to the non-visible spectrum image byaltering one or more local characteristics of the non-visible spectrumimage. In some cases, the local characteristics of the non-visibleobject that can be altered can be one or more of the following: (a) aModulation Transfer Function (MTF) of the non-visible spectrum image,(b) one or more histogram distributions of the non-visible spectrumimage, or (c) a change in a hue of the non-visible spectrum image, inthe case of a color non-visible spectrum image. In some cases, thehistogram distributions that can be altered to apply the haze to thenon-visible spectrum image can include at least one of: a brightnesshistogram distribution or a contrast histogram distribution.

In some cases, the depth cues that are applied to the non-visiblespectrum image can provide perspective. For example, the depth cues caninclude grid lines that are overlaid on the non-visible spectrum image.

Attention is now drawn to FIG. 3 , a flowchart illustrating a secondexample of a sequence of operations for enhancing a depth perception ofa non-visible spectrum image of a scene, in accordance with thepresently disclosed subject matter.

In accordance with the presently disclosed subject matter, the at leastone non-visible spectrum sensor 110 can be configured to capture anon-visible spectrum image of a scene at a capture time, the non-visiblespectrum image including one or more objects (block 304).

Processing circuitry 130 can be configured, e.g. using depth perceptionenhancement module 140, to classify one or more of the objects withoutderiving three-dimensional (3D) data from the non-visible spectrumimage, giving rise to one or more classified objects (block 308).

In some cases, the classified objects can be high priority objects inthe scene. For example, if the non-visible spectrum sensor 110 ismounted on an airborne platform (e.g., airplane, helicopter, drone,etc.) flying along a flight path, the classified objects in the scenecan be objects that are present along the flight path of the airborneplatform.

Processing circuitry 130 can be further configured, e.g. using depthperception enhancement module 140, to generate one or more depth cuesbased on one or more parameters associated with the classified objects(block 312). The parameters associated with a given classified object ofthe classified objects can be, for example, a relation between alocation (e.g., position and orientation) of the non-visible spectrumsensor 110 and the classified object (being indicative of a number ofpixels in the non-visible spectrum image that are associated with theclassified object), and an estimated height of the given classifiedobject.

Processing circuitry 130 can be configured. e.g. using depth perceptionenhancement module 140, to apply the depth cues to the non-visiblespectrum image to generate an enhanced depth perception image having anenhanced depth perception relative to the non-visible spectrum image(block 316).

In some cases, processing circuitry 130 can be configured to record thenon-visible spectrum image to provide a recorded non-visible spectrumimage, and to apply at least one of the depth cues to the recordednon-visible spectrum image, thereby giving rise to the enhanced depthperception image.

In some cases, the depth cues can be applied to the non-visible spectrumimage by overlaying the non-visible spectrum image with the depth cues(rather than modifying the non-visible spectrum image with the depthcues). For example, the depth cues can be optically injected by asynthetic image source (not shown) and combined with an unalterednon-visible spectrum image (i.e., overlaid on the non-visible spectrumimage), e.g. by an optical combiner, to display an enhanced depthperception image to the user of the system 100, as detailed furtherherein, inter alia with reference to FIGS. 4 to 7 .

In some cases, the depth cues can be overlaid onto the non-visiblespectrum image within an eyepiece of an optical instrument that displaysthe enhanced depth perception image to the user of the system 100, asdetailed further herein, inter alia with reference to FIGS. 4 to 7 . Insome cases, the optical instrument can be a night vision opticalinstrument.

Processing circuitry 130 can also be configured, e.g. using depthperception enhancement module 140, to display the enhanced depthperception image (block 320).

In some cases, the depth cues that are applied to (e.g., overlaid on ormerged with) the non-visible spectrum image can include one or moreshadows that are generated based on the classified objects, the shadowsbeing configured to at least one of: (a) vary a brightness of pixelswithin the enhanced depth perception image relative to the non-visiblespectrum image or (b) vary a color of pixels within the enhanced depthperception image relative to the non-visible spectrum image. In somecases, at least some of the shadows can be generated by one or morevirtual light sources. Additionally, or alternatively, in some cases, atleast some of the shadows can be generated based on a known position andorientation of an existing light source (sun, moon, other light source,etc.) that illuminates the scene that is captured by the non-visiblespectrum sensor 110.

In some cases, in which at least some of the shadows are generated byone or more virtual light sources, at least some of the virtual lightsources can be selected by the processing circuitry 130, based on theclassified objects. In some cases, processing circuitry 130 can furtherbe configured to define at least some of the parameters of a respectivevirtual light source, of the virtual light sources that it selects,based on at least one of the parameters associated with the classifiedobjects. Additionally. or alternatively, in some cases, a user candefine at least some of the parameters of a respective virtual lightsource, of the virtual light sources that are selected by processingcircuitry 130, based on at least one of the parameters associated withthe classified objects. The parameters of a respective virtual lightsource can be, for example, a magnitude, a position and an orientation,and/or other characteristics of the respective virtual light source.

In some cases, in which at least some of the shadows are generated byone or more virtual light sources, at least some of the virtual lightsources can be selected by a user, based on the classified objects. Theuser can also define the parameters of the virtual light sources thathe/she selects, based on at least one of the parameters associated withthe classified objects.

In some cases, the depth cues that are applied to the non-visiblespectrum image (e.g., overlaid on the non-visible spectrum image) caninclude one or more virtual objects, the virtual objects being based ona corresponding one or more physical objects that are of a known size.Prior to applying the virtual objects to (e.g., overlaying the virtualobjects on) the non-visible spectrum image, an actual size of each ofthe virtual objects can be determined, e.g. by processing circuitry 130,in accordance with a known size of the physical object to which itcorresponds, and a distance between the non-visible spectrum sensor 110and a location within the non-visible spectrum image at which therespective virtual object is to be applied (i.e., overlaid).

In some cases, processing circuitry 130 can be configured to define oneor more parameters that are associated with the virtual objects.Additionally, or alternatively, in some cases, a user can define one ormore parameters that are associated with the virtual objects. Theparameters that are associated with the virtual objects can be, forexample, a type(s) (e.g., car, building, tree, stationary, moving,etc.), a number, a location and/or a distribution of the virtualobjects.

Moreover, processing circuitry 130 can be configured to display thevirtual objects in the displayed enhanced depth perception image in amanner that distinguishes the virtual objects from real objects in thedisplayed enhanced depth perception image.

In some cases, the depth cues that are applied to (e.g., overlaid on)the non-visible spectrum image can include haze. In some cases,processing circuitry 130 can be configured to apply the haze to thenon-visible spectrum image by altering one or more local characteristicsof the non-visible spectrum image. In some cases, the localcharacteristics of the non-visible object that can be altered can be oneor more of the following: (a) a Modulation Transfer Function (MTF) ofthe non-visible spectrum image, (b) one or more histogram distributionsof the non-visible spectrum image, or (c) a change in a hue of thenon-visible spectrum image, in the case of a color non-visible spectrumimage. In some cases, the histogram distributions that can be altered toapply the haze to the non-visible spectrum image can include at leastone of: a brightness histogram distribution or a contrast histogramdistribution.

In some cases, the depth cues that are applied to the non-visiblespectrum image can provide perspective. For example, the depth cues caninclude grid lines that are overlaid on the non-visible spectrum image.

It is to be noted that, with reference to FIGS. 2 and 3 , some of theblocks can be integrated into a consolidated block or can be broken downto a few blocks and/or other blocks may be added. Furthermore, in somecases, the blocks can be performed in a different order than describedherein. It is to be further noted that some of the blocks are optional.It should be also noted that whilst the flow diagram is described alsowith reference to the system elements that realizes them, this is by nomeans binding, and the blocks can be performed by elements other thanthose described herein.

Attention is now drawn to FIG. 4 , a schematic diagram illustrating aschematic optical instrument 400 for displaying an enhanced depthperception image of the scene, in accordance with the presentlydisclosed subject matter.

In accordance with the presently disclosed subject matter, opticalinstrument 400 can be configured to include a regular eyepiece 410A andan enhanced eyepiece 410B. Enhanced eyepiece 410B can be configured tooverlay the non-visible spectrum image with depth cues, as detailedfurther herein, inter alia with reference to FIG. 7 .

In some cases, enhanced eyepiece 410B can be configured to include asynthetic image source 420. Synthetic image source 420 can be configuredto inject the depth cues to be overlaid on the non-visible spectrumimage. In some cases, synthetic image source 420 can include a liquidcrystal display (LCD) (not shown) and processing circuitry (not shown).In some cases, as illustrated in FIG. 4 , synthetic image source 420 canbe connected, via cable 430, to a power source 440.

Attention is now drawn to FIG. 5 , a schematic diagram illustrating anexploded view of an enhanced eyepiece 410B of the schematic opticalinstrument 400, in accordance with the presently disclosed subjectmatter.

In accordance with the presently disclosed subject matter, enhancedeyepiece 410B can be configured to include a synthetic image source 420.In some cases, synthetic image source 420 can be configured to include acasing 502, a backlight unit 504, and a liquid crystal display (LCD)506. Power source 444) can be configured to feed the backlight unit 504and the LCD 506 via cable 430. Mechanically, coupling means 512 can beconfigured to connect enhanced eyepiece 410B to an objective side of theoptical instrument 400.

In some cases, enhanced eyepiece 410B can be configured to include adiopter setting ring 514. Diopter setting ring 514 can be configured toset a diopter of an adjustable lens within the enhanced eyepiece 410Bfor both the depth cues and the non-visible sensor image.

Attention is now drawn to FIG. 6 , a schematic diagram illustratinganother perspective of the schematic optical instrument 400, inaccordance with the presently disclosed subject matter.

In accordance with the presently disclosed subject matter, schematicoptical instrument 400 can be configured to include a regular eyepiece410A and an enhanced eyepiece 410B. Enhanced eyepiece 410B can beconfigured to include a synthetic image source 420, coupling means 512,and a diopter setting ring 514, as detailed earlier herein, inter aliawith reference to FIG. 5 . Synthetic image source 420 can be connected,via cable 430, to a power source 440, as detailed earlier herein, interalia with reference to FIG. 5 .

Attention is now drawn to FIG. 7 , an optical diagram 700 illustratingoptical components of the enhanced eyepiece 410B, in accordance with thepresently disclosed subject matter.

In accordance with the presently disclosed subject matter, in somecases, enhanced eyepiece 410B can be configured to include anobjective-side lens 720; an observer-side lens 760; and an opticalcombiner 730 positioned between the objective-side lens 720 and theobserver-side lens 760. Objective-side lens 720 can be configured toobtain the non-visible spectrum image of the scene 710. Optical combiner730 can be configured to transfer the non-visible spectrum image fromthe objective-side lens 720 to the observer-side lens 760. Moreover,optical combiner 730 can be configured to reflect the depth cues (e.g.,generated by a synthetic image source 420) towards the observer-sidelens 760. Synthetic image source 420 can be configured to include animage emitting source 750 for emitting the depth cues, as illustrated inFIG. 7 . In some cases, a diopter setting ring 514 can be configured toset a diopter of an adjustable lens within the enhanced eyepiece 410B toaccurately overlay the non-visible sensor image with the depth cues. Theoverlaying of the non-visible sensor image with the depth cues resultsin an enhanced depth perception image being displayed to the user (at770).

In some cases, system 100 can be configured to record the non-visiblespectrum image that is output by the objective-side lens 720, using avideo recorder (not shown).

It is to be understood that the presently disclosed subject matter isnot limited in its application to the details set forth in thedescription contained herein or illustrated in the drawings. Thepresently disclosed subject matter is capable of other embodiments andof being practiced and carried out in various ways. Hence, it is to beunderstood that the phraseology and terminology employed herein are forthe purpose of description and should not be regarded as limiting. Assuch, those skilled in the art will appreciate that the conception uponwhich this disclosure is based may readily be utilized as a basis fordesigning other structures, methods, and systems for carrying out theseveral purposes of the present presently disclosed subject matter.

It will also be understood that the system according to the presentlydisclosed subject matter can be implemented, at least partly, as asuitably programmed computer. Likewise, the presently disclosed subjectmatter contemplates a computer program being readable by a computer forexecuting the disclosed method. The presently disclosed subject matterfurther contemplates a machine-readable memory tangibly embodying aprogram of instructions executable by the machine for executing thedisclosed method.

1. A method for providing depth perception to a two-dimensional (2D)representation of a given three-dimensional (3D) object within a 2Dnon-visible spectrum image of a scene, the method comprising: capturingthe 2D non-visible spectrum image at a capture time, by at least onenon-visible spectrum sensor; obtaining 3D data regarding the given 3Dobject independently of the 2D non-visible spectrum image; generatingone or more depth cues based on the 3D data; applying the depth cues onthe 2D representation to generate a depth perception image that providesthe depth perception to the 2D representation; and displaying the depthperception image.
 2. The method of claim 1, wherein the 3D data is apriori data regarding coordinates of a fixed coordinate systemestablished in space that are associated with the given 3D object, the apriori data being available prior to the capture time; and wherein thedepth cues are generated based on the a priori data and an actualposition and orientation of the non-visible spectrum sensor relative tothe fixed coordinate system at the capture time.
 3. The method of claim1, wherein the 3D data is one or more readings by an additional sensorthat is distinct from the non-visible spectrum sensor; and wherein thedepth cues are generated based on the readings and a first actualposition and orientation of the non-visible spectrum sensor at thecapture time relative to a second actual position and orientation of theadditional sensor at a second time of the readings.
 4. (canceled)
 5. Themethod of claim 1, further comprising: recording the 2D non-visiblespectrum image to provide a recording of the 2D non-visible spectrumimage; wherein the depth cues are applied on the 2D representationwithin the 2D non-visible spectrum image as recorded.
 6. (canceled) 7.(canceled)
 8. The method of claim 1, wherein the depth cues include oneor more of the following: (a) one or more shadows; (b) a virtual object;or (c) contour lines.
 9. The method of claim 8, wherein at least some ofthe shadows are generated by one or more virtual light sources.
 10. Themethod of claim 9, further comprising: selecting one or more selectedlight sources of the virtual light sources.
 11. (canceled)
 12. Themethod of claim 10, wherein, for at least one selected light source ofthe selected light sources, one or more parameters of the at least oneselected light source are defined by a user, the one or more parametersincluding a position and an orientation of the at least one selectedlight source.
 13. (canceled)
 14. (canceled)
 15. The method of claim 8,wherein the virtual object is distinguishable from the 2Drepresentation.
 16. A system for providing depth perception to atwo-dimensional (2D) representation of a given three-dimensional (3D)object within a 2D non-visible spectrum image of a scene, the systemcomprising: at least one non-visible spectrum sensor configured tocapture the 2D non-visible spectrum image at a capture time; and aprocessing circuitry configured to: obtain 3D data regarding the given3D object independently of the 2D non-visible spectrum image; generateone or more depth cues based on the 3D data; apply the depth cues on the2D representation to generate a depth perception image that provides thedepth perception to the 2D representation; and display the depthperception image.
 17. The system of claim 16, wherein the 3D data is apriori data regarding coordinates of a fixed coordinate systemestablished in space that are associated with the given 3D object, the apriori data being available prior to the capture time; and wherein thedepth cues are applied based on the a priori data and an actual positionand orientation of the non-visible spectrum sensor relative to the fixedcoordinate system at the capture time.
 18. The system of claim 16,wherein the 3D data is one or more readings by an additional sensor thatis distinct from the non-visible spectrum sensor; and wherein the depthcues are generated based on the readings and a first actual position andorientation of the non-visible spectrum sensor at the capture timerelative to a second actual position and orientation of the additionalsensor at a second time of the readings.
 19. (canceled)
 20. The systemof claim 16, wherein the processing circuitry is further configured to:record the 2D non-visible spectrum image to provide a recording of the2D non-visible spectrum image; wherein the depth cues are applied on the2D non-visible spectrum image as recorded.
 21. (canceled)
 22. (canceled)23. The system of claim 16, wherein the depth cues include one or moreof the following: (a) one or more shadows; (b) a virtual object; or (c)contour lines.
 24. The system of claim 23, wherein at least some of theshadows are generated by one or more virtual light sources.
 25. Thesystem of claim 24, wherein the processing circuitry is furtherconfigured to: select one or more selected light sources of the virtuallight sources.
 26. (canceled)
 27. The system of claim 25, wherein, forat least one selected light source of the selected light sources, one ormore parameters of the at least one selected light source are defined bya user of the system, the one or more parameters including a positionand an orientation of the at least one selected light source. 28.(canceled)
 29. (canceled)
 30. The system of claim 23, wherein thevirtual object is distinguishable from the 2D representation.
 31. Anon-transitory computer readable storage medium having computer readableprogram code embodied therewith, the computer readable program code,executable by processing circuitry of a computer to perform a method forproviding depth perception to a two-dimensional (2D) representation of agiven three-dimensional (3D) object within a 2D non-visible spectrumimage of a scene, the method comprising: capturing the 2D non-visiblespectrum image at a capture time, by at least one non-visible spectrumsensor; obtaining 3D data regarding the given 3D object independently ofthe 2D non-visible spectrum image; generating one or more depth cuesbased on the 3D data; applying the depth cues on the 2D representationto generate a depth perception image that provides the depth perceptionto the 2D representation; and displaying the depth perception image.